Automated Coding of Open-Ended Survey Responses

Background: Manual coding of open-ended survey responses is time-consuming, expensive, and error-prone. However, this task can be recast as a multi-label text classification problem, to which natural language processing and machine learning techniques can be applied. Aim: The aim of this work is to assess the suitability of various machine learning techniques to the problem of automated coding of open-ended survey responses, and to explore the use of techniques to automate the search for an appropriate model. Data: The data for this project consist of approximately 28,500 manually-coded responses by 2,000 respondents to open-ended survey questions from the 2008 American National Election Studies survey. Methods: We compare the application of two types of machine learning methods to this task: L1-regularized logistic regression, and recurrent neural networks. In an effort to provide a fair comparison, while minimizing bias in favor of novelty, we make use of Bayesian optimization to perform an automated search over the configuration space of these two types of models, and make use of a reusable holdout in an attempt to prevent overfitting. Results: In the case of a traditional logistic regression model, we find that Bayesian optimization is an effective way to do feature selection and hyperparameter tuning. For recurrent neural networks, which involve a much larger space of possible configurations, we have been unable to produce a consistent improvement over the baseline in performance on this data. Conclusions: Although we do not presently recommend recurrent neural networks as a effective or easy-to-use approach to this kind of limited-data multi-label classification problem, we believe that with further work, recurrent neural networks do have advantages which can be realized through an automated model selection process.